Concurrent set storage in distributed storage network

ABSTRACT

For each original data segment, a distributed storage processing unit generates encoded slices designed to prevent the original data segment from being reconstructed using fewer than a threshold number of encoded slices. Multiple encoded slices are generated for each of two different data segments, and the slices associated with the first and second data segment are stored substantially concurrently in different storage sets employing different distributed storage units. Encoded slices for even and odd data segments can be stored in different storage sets, or longer sequences of data segments can be stored in alternating storage sets. Storage sets can also be determined by the vault generation of a particular data segment.

CROSS REFERENCE TO RELATED PATENTS

This patent application is claiming priority under 35 USC §119(e) to aprovisionally filed patent application entitled “DISTRIBUTED STORAGENETWORK DATA ROUTING,” having a provisional filing date of Oct. 30, 2009and a provisional Ser. No. 61/256,419, filed Oct. 30, 2009, which isincorporated herein in its entirety by reference for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

NOT APPLICABLE

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

NOT APPLICABLE

BACKGROUND OF THE INVENTION

1. Technical Field of the Invention

This invention relates generally to computing systems and moreparticularly to data storage solutions within such computing systems.

2. Description of Related Art

Computers are known to communicate, process, and store data. Suchcomputers range from wireless smart phones to data centers that supportmillions of web searches, stock trades, or on-line purchases every day.In general, a computing system generates data and/or manipulates datafrom one form into another. For instance, an image sensor of thecomputing system generates raw picture data and, using an imagecompression program (e.g., JPEG, MPEG, etc.), the computing systemmanipulates the raw picture data into a standardized compressed image.

With continued advances in processing speed and communication speed,computers are capable of processing real time multimedia data forapplications ranging from simple voice communications to streaming highdefinition video. As such, general-purpose information appliances arereplacing purpose-built communications devices (e.g., a telephone). Forexample, smart phones can support telephony communications but they arealso capable of text messaging and accessing the internet to performfunctions including email, web browsing, remote applications access, andmedia communications (e.g., telephony voice, image transfer, musicfiles, video files, real time video streaming. etc.).

Each type of computer is constructed and operates in accordance with oneor more communication, processing, and storage standards. As a result ofstandardization and with advances in technology, more and moreinformation content is being converted into digital formats. Forexample, more digital cameras are now being sold than film cameras, thusproducing more digital pictures. As another example, web-basedprogramming is becoming an alternative to over the air televisionbroadcasts and/or cable broadcasts. As further examples, papers, books,video entertainment, home video, etc. are now being stored digitally,which increases the demand on the storage function of computers.

A typical computer storage system includes one or more memory devicesaligned with the needs of the various operational aspects of thecomputer's processing and communication functions. Generally, theimmediacy of access dictates what type of memory device is used. Forexample, random access memory (RAM) memory can be accessed in any randomorder with a constant response time, thus it is typically used for cachememory and main memory. By contrast, memory device technologies thatrequire physical movement such as magnetic disks, tapes, and opticaldiscs, have a variable response time as the physical movement can takelonger than the data transfer, thus they are typically used forsecondary memory (e.g., hard drive, backup memory, etc.).

A computer's storage system will be compliant with one or more computerstorage standards that include, but are not limited to, network filesystem (NFS), flash file system (FFS), disk file system (DFS), smallcomputer system interface (SCSI), internet small computer systeminterface (iSCSI), file transfer protocol (FTP), and web-baseddistributed authoring and versioning (WebDAV). These standards specifythe data storage format (e.g., files, data objects, data blocks,directories, etc.) and interfacing between the computer's processingfunction and its storage system, which is a primary function of thecomputer's memory controller.

Despite the standardization of the computer and its storage system,memory devices fail; especially commercial grade memory devices thatutilize technologies incorporating physical movement (e.g., a discdrive). For example, it is fairly common for a disc drive to routinelysuffer from bit level corruption and to completely fail after threeyears of use. One solution is to a higher-grade disc drive, which addssignificant cost to a computer.

Another solution is to utilize multiple levels of redundant disc drivesto replicate the data into two or more copies. One such redundant driveapproach is called redundant array of independent discs (RAID). In aRAID device, a RAID controller adds parity data to the original databefore storing it across the array. The parity data is calculated fromthe original data such that the failure of a disc will not result in theloss of the original data. For example, RAID 5 uses three discs toprotect data from the failure of a single disc. The parity data, andassociated redundancy overhead data, reduces the storage capacity ofthree independent discs by one third (e.g., n−1=capacity). RAID 6 canrecover from a loss of two discs and requires a minimum of four discswith a storage capacity of n−2.

While RAID addresses the memory device failure issue, it is not withoutits own failures issues that affect its effectiveness, efficiency andsecurity. For instance, as more discs are added to the array, theprobability of a disc failure increases, which increases the demand formaintenance. For example, when a disc fails, it needs to be manuallyreplaced before another disc fails and the data stored in the RAIDdevice is lost. To reduce the risk of data loss, data on a RAID deviceis typically copied on to one or more other RAID devices. While thisaddresses the loss of data issue, it raises a security issue sincemultiple copies of data are available, which increases the chances ofunauthorized access. Further, as the amount of data being stored grows,the overhead of RAID devices becomes a non-trivial efficiency issue.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 is a schematic block diagram of an embodiment of a computingsystem in accordance with the invention;

FIG. 2 is a schematic block diagram of an embodiment of a computing corein accordance with the invention;

FIG. 3 is a schematic block diagram of an embodiment of a distributedstorage processing unit in accordance with the invention;

FIG. 4 is a schematic block diagram of an embodiment of a grid module inaccordance with the invention;

FIG. 5 is a diagram of an example embodiment of error coded data slicecreation in accordance with the invention;

FIG. 6 is a schematic block diagram of another embodiment of a computingsystem in accordance with the invention;

FIG. 7 is a flowchart illustrating the retrieval of distributedly storeddata;

FIG. 8 is a schematic block diagram of another embodiment of a computingsystem in accordance with the invention;

FIG. 9 is a schematic block diagram of an embodiment of a distributedstorage (DS) unit in accordance with the invention;

FIG. 10 is a flowchart illustrating the storing of distributedly storeddata;

FIG. 11 is a schematic block diagram of another embodiment of acomputing system in accordance with the invention;

FIG. 12 is a block diagram of an embodiment of layered message creationin accordance with the invention;

FIG. 13 is a flowchart illustrating the creation of a layered message;and

FIG. 14 is a flowchart illustrating the processing of a layered message.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a schematic block diagram of a computing system 10 thatincludes one or more of a first type of user devices 12, one or more ofa second type of user devices 14, at least one distributed storage (DS)processing unit 16, at least one DS managing unit 18, at least onestorage integrity processing unit 20, and a distributed storage network(DSN) memory 22 coupled via a network 24. The network 24 may include oneor more wireless and/or wire lined communication systems; one or moreprivate intranet systems and/or public internet systems; and/or one ormore local area networks (LAN) and/or wide area networks (WAN).

The DSN memory 22 includes a plurality of distributed storage (DS) units36 for storing data of the system. Each of the DS units 36 includes aprocessing module and memory and may be located at a geographicallydifferent site than the other DS units (e.g., one in Chicago, one inMilwaukee, etc.). The processing module may be a single processingdevice or a plurality of processing devices. Such a processing devicemay be a microprocessor, micro-controller, digital signal processor,microcomputer, central processing unit, field programmable gate array,programmable logic device, state machine, logic circuitry, analogcircuitry, digital circuitry, and/or any device that manipulates signals(analog and/or digital) based on hard coding of the circuitry and/oroperational instructions. The processing module may have an associatedmemory and/or memory element, which may be a single memory device, aplurality of memory devices, and/or embedded circuitry of the processingmodule. Such a memory device may be a read-only memory, random accessmemory, volatile memory, non-volatile memory, static memory, dynamicmemory, flash memory, cache memory, and/or any device that storesdigital information. Note that if the processing module includes morethan one processing device, the processing devices may be centrallylocated (e.g., directly coupled together via a wired and/or wireless busstructure) or may be distributedly located (e.g., cloud computing viaindirect coupling via a local area network and/or a wide area network).Further note that when the processing module implements one or more ofits functions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element stores, and the processing module executes,hard coded and/or operational instructions corresponding to at leastsome of the steps and/or functions illustrated in FIGS. 1-14.

Each of the user devices 12-14, the DS processing unit 16, the DSmanaging unit 18, and the storage integrity processing unit 20 may be aportable computing device (e.g., a social networking device, a gamingdevice, a cell phone, a smart phone, a personal digital assistant, adigital music player, a digital video player, a laptop computer, ahandheld computer, a video game controller, and/or any other portabledevice that includes a computing core) and/or a fixed computing device(e.g., a personal computer, a computer server, a cable set-top box, asatellite receiver, a television set, a printer, a fax machine, homeentertainment equipment, a video game console, and/or any type of homeor office computing equipment). Such a portable or fixed computingdevice includes a computing core 26 and one or more interfaces 30, 32,and/or 33. An embodiment of the computing core 26 will be described withreference to FIG. 2.

With respect to the interfaces, each of the interfaces 30, 32, and 33includes software and/or hardware to support one or more communicationlinks via the network 24 and/or directly. For example, interfaces 30support a communication link (wired, wireless, direct, via a LAN, viathe network 24, etc.) between the first type of user device 14 and theDS processing unit 16. As another example, DSN interface 32 supports aplurality of communication links via the network 24 between the DSNmemory 22 and the DS processing unit 16, the first type of user device12, and/or the storage integrity processing unit 20. As yet anotherexample, interface 33 supports a communication link between the DSmanaging unit 18 and any one of the other devices and/or units 12, 14,16, 20, and/or 22 via the network 24.

In general and with respect to data storage, the system 10 supportsthree primary functions: distributed network data storage management,distributed data storage and retrieval, and data storage integrityverification. In accordance with these three primary functions, data canbe distributedly stored in a plurality of physically different locationsand subsequently retrieved in a reliable and secure manner regardless offailures of individual storage devices, failures of network equipment,the duration of storage, the amount of data being stored, attempts athacking the data, etc.

The DS managing unit 18 performs distributed network data storagemanagement functions, which include establishing distributed datastorage parameters, performing network operations, performing networkadministration, and/or performing network maintenance. The DS managingunit 18 establishes the distributed data storage parameters (e.g.,allocation of virtual DSN memory space, distributed storage parameters,security parameters, billing information, user profile information,etc.) for one or more of the user devices 12-14 (e.g., established forindividual devices, established for a user group of devices, establishedfor public access by the user devices, etc.). For example, the DSmanaging unit 18 coordinates the creation of a vault (e.g., a virtualmemory block) within the DSN memory 22 for a user device (for a group ofdevices, or for public access). The DS managing unit 18 also determinesthe distributed data storage parameters for the vault. In particular,the DS managing unit 18 determines a number of slices (e.g., the numberthat a data segment of a data file and/or data block is partitioned intofor distributed storage) and a read threshold value (e.g., the minimumnumber of slices required to reconstruct the data segment).

As another example, the DS managing module 18 creates and stores,locally or within the DSN memory 22, user profile information. The userprofile information includes one or more of authentication information,permissions, and/or the security parameters. The security parameters mayinclude one or more of encryption/decryption scheme, one or moreencryption keys, key generation scheme, and data encoding/decodingscheme.

As yet another example, the DS managing unit 18 creates billinginformation for a particular user, user group, vault access, publicvault access, etc. For instance, the DS managing unit 18 tracks thenumber of times user accesses a private vault and/or public vaults,which can be used to generate a per-access bill. In another instance,the DS managing unit 18 tracks the amount of data stored and/orretrieved by a user device and/or a user group, which can be used togenerate a per-data-amount bill.

The DS managing unit 18 also performs network operations, networkadministration, and/or network maintenance. As at least part ofperforming the network operations and/or administration, the DS managingunit 18 monitors performance of the devices and/or units of the system10 for potential failures, determines the devices and/or unit'sactivation status, determines the devices' and/or units' loading, andany other system level operation that affects the performance level ofthe system 10. For example, the DS managing unit 18 receives andaggregates network management alarms, alerts, errors, statusinformation, performance information, and messages from the devices12-14 and/or the units 16, 20, 22. For example, the DS managing unit 18receives a simple network management protocol (SNMP) message regardingthe status of the DS processing unit 16.

The DS managing unit 18 performs the network maintenance by identifyingequipment within the system 10 that needs replacing, upgrading,repairing, and/or expanding. For example, the DS managing unit 18determines that the DSN memory 22 needs more DS units 36 or that one ormore of the DS units 36 needs updating.

The second primary function (i.e., distributed data storage andretrieval) begins and ends with a user device 12-14. For instance, if asecond type of user device 14 has a data file 38 and/or data block 40 tostore in the DSN memory 22, it send the data file 38 and/or data block40 to the DS processing unit 16 via its interface 30. As will bedescribed in greater detail with reference to FIG. 2, the interface 30functions to mimic a conventional operating system (OS) file systeminterface (e.g., network file system (NFS), flash file system (FFS),disk file system (DFS), file transfer protocol (FTP), web-baseddistributed authoring and versioning (WebDAV), etc.) and/or a blockmemory interface (e.g., small computer system interface (SCSI), internetsmall computer system interface (iSCSI), etc.). In addition, theinterface 30 may attach a user identification code (ID) to the data file38 and/or data block 40.

The DS processing unit 16 receives the data file 38 and/or data block 40via its interface 30 and performs a distributed storage (DS) process 34thereon (e.g., an error coding dispersal storage function). The DSprocessing 34 begins by partitioning the data file 38 and/or data block40 into one or more data segments, which is represented as Y datasegments. For example, the DS processing 34 may partition the data file38 and/or data block 40 into a fixed byte size segment (e.g., 2¹ to2^(n) bytes, where n=>2) or a variable byte size (e.g., change byte sizefrom segment to segment, or from groups of segments to groups ofsegments, etc.).

For each of the Y data segments, the DS processing 34 error encodes(e.g., forward error correction (FEC), information dispersal algorithm,or error correction coding) and slices (or slices then error encodes)the data segment into a plurality of error coded (EC) data slices 42-48,which is represented as X slices per data segment. The number of slices(X) per segment, which corresponds to a number of pillars n, is set inaccordance with the distributed data storage parameters and the errorcoding scheme. For example, if a Reed-Solomon (or other FEC scheme) isused in an n/k system, then a data segment is divided into n slices,where k number of slices is needed to reconstruct the original data(i.e., k is the threshold). As a few specific examples, the n/k factormay be 5/3; 6/4; 8/6; 8/5; 16/10.

For each slice 42-48, the DS processing unit 16 creates a unique slicename and appends it to the corresponding slice 42-48. The slice nameincludes universal DSN memory addressing routing information (e.g.,virtual memory addresses in the DSN memory 22) and user-specificinformation (e.g., user ID, file name, data block identifier, etc.).

The DS processing unit 16 transmits the plurality of EC slices 42-48 toa plurality of DS units 36 of the DSN memory 22 via the DSN interface 32and the network 24. The DSN interface 32 formats each of the slices fortransmission via the network 24. For example, the DSN interface 32 mayutilize an internet protocol (e.g., TCP/IP, etc.) to packetize theslices 42-48 for transmission via the network 24.

The number of DS units 36 receiving the slices 42-48 is dependent on thedistributed data storage parameters established by the DS managing unit18. For example, the DS managing unit 18 may indicate that each slice isto be stored in a different DS unit 36. As another example, the DSmanaging unit 18 may indicate that like slice numbers of different datasegments are to be stored in the same DS unit 36. For example, the firstslice of each of the data segments is to be stored in a first DS unit36, the second slice of each of the data segments is to be stored in asecond DS unit 36, etc. In this manner, the data is encoded anddistributedly stored at physically diverse locations to improved datastorage integrity and security. Further examples of encoding the datasegments will be provided with reference to one or more of FIGS. 2-9.

Each DS unit 36 that receives a slice 42-48 for storage translates thevirtual DSN memory address of the slice into a local physical addressfor storage. Accordingly, each DS unit 36 maintains a virtual tophysical memory mapping to assist in the storage and retrieval of data.

The first type of user device 12 performs a similar function to storedata in the DSN memory 22 with the exception that it includes the DSprocessing. As such, the device 12 encodes and slices the data fileand/or data block it has to store. The device then transmits the slices35 to the DSN memory via its DSN interface 32 and the network 24.

For a second type of user device 14 to retrieve a data file or datablock from memory, it issues a read command via its interface 30 to theDS processing unit 16. The DS processing unit 16 performs the DSprocessing 34 to identify the DS units 36 storing the slices of the datafile and/or data block based on the read command. The DS processing unit16 may also communicate with the DS managing unit 18 to verify that theuser device 14 is authorized to access the requested data.

Assuming that the user device is authorized to access the requesteddata, the DS processing unit 16 issues slice read commands to at least athreshold number of the DS units 36 storing the requested data (e.g., toat least 10 DS units for a 16/10 error coding scheme). Each of the DSunits 36 receiving the slice read command, verifies the command,accesses its virtual to physical memory mapping, retrieves the requestedslice, or slices, and transmits it to the DS processing unit 16.

Once the DS processing unit 16 has received a read threshold number ofslices for a data segment, it performs an error decoding function andde-slicing to reconstruct the data segment. When Y number of datasegments has been reconstructed, the DS processing unit 16 provides thedata file 38 and/or data block 40 to the user device 14. Note that thefirst type of user device 12 performs a similar process to retrieve adata file and/or data block.

The storage integrity processing unit 20 performs the third primaryfunction of data storage integrity verification. In general, the storageintegrity processing unit 20 periodically retrieves slices 45, and/orslice names, of a data file or data block of a user device to verifythat one or more slices have not been corrupted or lost (e.g., the DSunit failed). The retrieval process mimics the read process previouslydescribed.

If the storage integrity processing unit 20 determines that one or moreslices is corrupted or lost, it rebuilds the corrupted or lost slice(s)in accordance with the error coding scheme. The storage integrityprocessing unit 20 stores the rebuild slice, or slices, in theappropriate DS unit(s) 36 in a manner that mimics the write processpreviously described.

FIG. 2 is a schematic block diagram of an embodiment of a computing core26 that includes a processing module 50, a memory controller 52, mainmemory 54, a video graphics processing unit 55, an input/output (TO)controller 56, a peripheral component interconnect (PCI) interface 58,at least one 10 device interface module 62, a read only memory (ROM)basic input output system (BIOS) 64, and one or more memory interfacemodules. The memory interface module(s) includes one or more of auniversal serial bus (USB) interface module 66, a host bus adapter (HBA)interface module 68, a network interface module 70, a flash interfacemodule 72, a hard drive interface module 74, and a DSN interface module76. Note the DSN interface module 76 and/or the network interface module70 may function as the interface 30 of the user device 14 of FIG. 1.Further note that the 10 device interface module 62 and/or the memoryinterface modules may be collectively or individually referred to as IOports.

The processing module 50 may be a single processing device or aplurality of processing devices. Such a processing device may be amicroprocessor, micro-controller, digital signal processor,microcomputer, central processing unit, field programmable gate array,programmable logic device, state machine, logic circuitry, analogcircuitry, digital circuitry, and/or any device that manipulates signals(analog and/or digital) based on hard coding of the circuitry and/oroperational instructions. The processing module 50 may have anassociated memory and/or memory element, which may be a single memorydevice, a plurality of memory devices, and/or embedded circuitry of theprocessing module 50. Such a memory device may be a read-only memory,random access memory, volatile memory, non-volatile memory, staticmemory, dynamic memory, flash memory, cache memory, and/or any devicethat stores digital information. Note that if the processing module 50includes more than one processing device, the processing devices may becentrally located (e.g., directly coupled together via a wired and/orwireless bus structure) or may be distributedly located (e.g., cloudcomputing via indirect coupling via a local area network and/or a widearea network). Further note that when the processing module 50implements one or more of its functions via a state machine, analogcircuitry, digital circuitry, and/or logic circuitry, the memory and/ormemory element storing the corresponding operational instructions may beembedded within, or external to, the circuitry comprising the statemachine, analog circuitry, digital circuitry, and/or logic circuitry.Still further note that, the memory element stores, and the processingmodule 50 executes, hard coded and/or operational instructionscorresponding to at least some of the steps and/or functions illustratedin FIGS. 1-14.

FIG. 3 is a schematic block diagram of an embodiment of a dispersedstorage (DS) processing module 34 of user device 12 and/or of the DSprocessing unit 16. The DS processing module 34 includes a gatewaymodule 78, an access module 80, a grid module 82, and a storage module84. The DS processing module 34 may also include an interface 30 and theDSnet interface 32 or the interfaces 68 and/or 70 may be part of user 12or of the DS processing unit 14. The DS processing module 34 may furtherinclude a bypass/feedback path between the storage module 84 to thegateway module 78.

In an example of storing data, the gateway module 78 receives anincoming data object (e.g., a data file, a data block, an EC data slice,etc.) that includes a user ID field 86, an object name field 88, and thedata field 40. The gateway module 78 authenticates the user associatedwith the data object by verifying the user ID 86 with the managing unit18 and/or another authenticating unit. When the user is authenticated,the gateway module 78 obtains user information from the management unit18, the user device, and/or the other authenticating unit. The userinformation includes a vault identifier, operational parameters, anduser attributes (e.g., user data, billing information, etc.). A vaultidentifier identifies a vault, which is a virtual memory space that mapsto a set of DS storage units 36. For example, vault 1 (i.e., user 1'sDSN memory space) includes eight DS storage units (X=8 wide) and vault 2(i.e., user 2's DSN memory space) includes sixteen DS storage units(X=16 wide). The operational parameters may include an error codingalgorithm, the width n (number of pillars X or slices per segment forthis vault), a read threshold T, an encryption algorithm, a slicingparameter, a compression algorithm, an integrity check method, cachingsettings, parallelism settings, and/or other parameters that may be usedto access the DSN memory layer.

The gateway module uses the user information to assign a source name tothe data. For instance, the gateway module 60 determines the source nameof the data object 40 based on the vault identifier and the data object.For example, the source name may contain a data name (block number or afile number), the vault generation number, the reserved field, and thevault identifier. The data name may be randomly assigned but isassociated with the user data object.

The access module 62 receives the data object 40 and creates a series ofdata segments 1 through Y 90-92 therefrom. The number of segments Y maybe chosen or randomly assigned based on a selected segment size and thesize of the data object. For example, if the number of segments ischosen to be a fixed number, then the size of the segments varies as afunction of the size of the data object. For instance, if the dataobject is an image file of 4,194,304 eight bit bytes (e.g., 33,554,432bits) and the number of segments Y=131,072, then each segment is 256bits or 32 bytes. As another example, if segment sized is fixed, thenthe number of segments Y varies based on the size of data object. Forinstance, if the data object is an image file of 4,194,304 bytes and thefixed size of each segment is 4,096 bytes, the then number of segmentsY=1,024. Note that each segment is associated with the source name.

The grid module 82 may pre-manipulate (e.g., compression, encryption,cyclic redundancy check (CRC), etc.) each of the data segments beforeperforming an error coding function of the error coding dispersalstorage function to produce a pre-manipulated data segment. The gridmodule 82 then error encodes (e.g., Reed-Solomon, Convolution encoding,Trellis encoding, etc.) the data segment or pre-manipulated data segmentinto X error coded data slices 42-44. The grid module 64 determines aunique slice name for each error coded data slice and attaches it to thedata slice.

In some embodiments, the slice name includes a universal routinginformation field and a vault specific field. In an embodiment, theuniversal routing information field is 24 bytes and the vault specificfield is 24 bytes. The universal routing information field contains aslice index, the vault ID, the vault generation, and the reserved field.The slice index is based on the pillar number n and the vault ID suchthat it is unique for each pillar (e.g., slices of the same pillar forthe same vault for any segment will share the same slice index). Thevault specific field contains a data name that may include the file IDand a segment number (e.g., a sequential numbering of the data segmentsof a simple data object or a data block number).

The data name field may be the same for slice names of slices for thesame data segment and may vary for slice names of different datasegments. The file ID portion of data name may not vary for any slicename of the same data object. Note that the DS processing module 34 maymodify the data name field such that the file ID is not transparent(e.g., produce a data name from a hash of the source name to disguisethe file ID).

The value X, or the number of pillars (e.g., X=16), is chosen as aparameter of the error coding dispersal storage function. Otherparameters of the error coding dispersal function include a readthreshold T, a write threshold W, etc. The read threshold (e.g., T=10,when X=16) corresponds to the minimum number of error-free error codeddata slices required to reconstruct the data segment. In other words,the DS processing module 34 can compensate for X−T (e.g., 16−10=6)missing error coded data slices per data segment. The write threshold Wcorresponds to a minimum number of DS storage units that acknowledgeproper storage of their respective data slices before the DS processingmodule indicates proper storage of the encoded data segment. Note thatthe write threshold is greater than or equal to the read threshold for agiven number of pillars (X).

The grid module 82 also determines which of the DS storage units 36 willstore the EC data slices based on a dispersed storage memory mappingassociated with the user's vault and/or DS storage unit 36 attributes.The DS storage unit attributes includes availability, self-selection,performance history, link speed, link latency, ownership, available DSNmemory, domain, cost, a prioritization scheme, a centralized selectionmessage from another source, a lookup table, data ownership, and/or anyother factor to optimize the operation of the computing system. Notethat the number of DS storage units 36 is equal to or greater than thenumber of pillars (e.g., X) so that no more than one error coded dataslice of the same data segment is stored on the same DS storage unit 36.Further note that EC data slices of the same pillar number but ofdifferent segments (e.g., EC data slice 1 of data segment 1 and EC dataslice 1 of data segment 2) may be stored on the same or different DSstorage units 36.

The storage module 84 performs an integrity check on the EC data slicesand, when successful, transmits the EC data slices 1 through X of eachsegment 1 through Y to the DS Storage units. Each of the DS storageunits 36 stores its EC data slice and keeps a table to convert thevirtual DSN address of the EC data slice into physical storageaddresses.

In an example of a read operation, the user device 12 and/or 14 sends aread request to the DS processing unit 14, which authenticates therequest. When the request is authentic, the DS processing unit 14 sendsa read message to each of the DS storage units 36 storing slices of thedata object being read. The slices are received via the DSnet interface32 and processed by the storage module 84, which performs a parity checkand provides the slices to the grid module 82 when the parity check wassuccessful. The grid module 82 decodes the slices in accordance with theerror coding dispersal storage function to reconstruct the data segment.The access module 80 reconstructs the data object from the data segmentsand the gateway module 78 formats the data object for transmission tothe user device.

FIG. 4 is a schematic block diagram of an embodiment of a grid module 82that includes a control unit 73, a pre-data manipulator 75, an encoder77, a slicer 79, a post-data manipulator 81, a pre-data de-manipulator83, a decoder 85, a de-slicer 87, and/or a post-data de-manipulator 89.Note that the control unit 73 may be partially or completely external tothe grid module 82. For example, the control unit 73 may be part of thecomputing core at a remote location, part of a user device, part of theDS managing unit 18, or distributed amongst one or more DS storageunits.

In an example of write operation, the pre-data manipulator 75 receives adata segment 90-92 and a write instruction from an authorized userdevice. The pre-data manipulator 75 determines if pre-manipulation ofthe data segment 90-92 is required and, if so, what type. The pre-datamanipulator 75 may make the determination independently or based oninstructions from the control unit 73, where the determination is baseda computing system-wide predetermination, a table lookup, vaultparameters associated with the user identification, the type of data,security requirements, available DSN memory, performance requirements,and/or other metadata.

Once a positive determination is made, the pre-data manipulator 75manipulates the data segment 90-92 in accordance with the type ofmanipulation. For example, the type of manipulation may be compression(e.g., Lempel-Ziv-Welch, Huffman, Golomb, fractal, wavelet, etc.),signatures (e.g., Digital Signature Algorithm (DSA), Elliptic Curve DSA,Secure Hash Algorithm, etc.), watermarking, tagging, encryption (e.g.,Data Encryption Standard, Advanced Encryption Standard, etc.), addingmetadata (e.g., time/date stamping, user information, file type, etc.),cyclic redundancy check (e.g., CRC32), and/or other data manipulationsto produce the pre-manipulated data segment.

The encoder 77 encodes the pre-manipulated data segment 92 using aforward error correction (FEC) encoder (and/or other type of erasurecoding and/or error coding) to produce an encoded data segment 94. Theencoder 77 determines which forward error correction algorithm to usebased on a predetermination associated with the user's vault, a timebased algorithm, user direction, DS managing unit direction, controlunit direction, as a function of the data type, as a function of thedata segment 92 metadata, and/or any other factor to determine algorithmtype. The forward error correction algorithm may be Golay,Multidimensional parity, Reed-Solomon, Hamming, Bose Ray ChauduriHocquenghem (BCH), Cauchy-Reed-Solomon, or any other FEC encoder. Notethat the encoder 77 may use a different encoding algorithm for each datasegment 92, the same encoding algorithm for the data segments 92 of adata object, or a combination thereof.

The encoded data segment 94 is of greater size than the data segment 92by the overhead rate of the encoding algorithm by a factor of d*(X/T),where d is size of the data segment 92, X is the width or number ofslices, and T is the read threshold. In this regard, the correspondingdecoding process can accommodate at most X−T missing EC data slices andstill recreate the data segment 92. For example, if X=16 and T=10, thenthe data segment 92 will be recoverable as long as 10 or more EC dataslices per segment are not corrupted.

The slicer 79 transforms the encoded data segment 94 into EC data slicesin accordance with the slicing parameter from the vault for this userand/or data segment 92. For example, if the slicing parameter is X=16,then the slicer slices each encoded data segment 94 into 16 encodedslices.

The post-data manipulator 81 performs, if enabled, post-manipulation onthe encoded slices to produce the EC data slices. If enabled, thepost-data manipulator 81 determines the type of post-manipulation, whichmay be based on a computing system-wide predetermination, parameters inthe vault for this user, a table lookup, the user identification, thetype of data, security requirements, available DSN memory, performancerequirements, control unit directed, and/or other metadata. Note thatthe type of post-data manipulation may include slice level compression,signatures, encryption, CRC, addressing, watermarking, tagging, addingmetadata, and/or other manipulation to improve the effectiveness of thecomputing system.

In an example of a read operation, the post-data de-manipulator 89receives at least a read threshold number of EC data slices and performsthe inverse function of the post-data manipulator 81 to produce aplurality of encoded slices. The de-slicer 87 de-slices the encodedslices to produce an encoded data segment 94. The decoder 85 performsthe inverse function of the encoder 77 to recapture the data segment90-92. The pre-data de-manipulator 83 performs the inverse function ofthe pre-data manipulator 75 to recapture the data segment.

FIG. 5 is a diagram of an example of slicing an encoded data segment 94by the slicer 79. In this example, the encoded data segment includesthirty-two bits, but may include more or less bits. The slicer 79disperses the bits of the encoded data segment 94 across the EC dataslices in a pattern as shown. As such, each EC data slice does notinclude consecutive bits of the data segment 94 reducing the impact ofconsecutive bit failures on data recovery. For example, if EC data slice2 (which includes bits 1, 5, 9, 13, 17, 25, and 29) is unavailable(e.g., lost, inaccessible, or corrupted), the data segment can bereconstructed from the other EC data slices (e.g., 1, 3 and 4 for a readthreshold of 3 and a width of 4).

FIG. 6 is a schematic block diagram of another embodiment of a computingsystem 102 that may provide access to slices from a cache memory inaddition to a DSN memory. The computing system 102 includes a pluralityof user devices 1-U, a DS processing unit 16, a cache memory 103, andthe DSN memory 22.

One of the user devices 1-U may from time to time request retrieval of adata object by sending a retrieval request message to the DS processingunit 16. The DS processing unit 16 determines where to retrieve theslices to reconstruct the data object. The slices may be located in thecache memory 103. The DS processing unit 16 may have previously storedthe slices in the cache memory 103. In another embodiment, at least twoDS processing units may communicate with each other to locate andretrieve slices stored in cache memory 103. Note that more than onecache memory may be utilized in the system.

The cache memory 103 includes a slice memory 104 and a distributed hashtable (DHT) 106, and may be implemented with one or more of a magnetichard disk, NAND flash, read only memory, optical disk, and/or any othertype of read-only, or read/write memory. In an embodiment, the cachememory 103 may be implemented as part of the DS processing unit.

The slice memory 104 stores EC data slices received as slices from theDS processing unit 16. The slice memory sends the slices to the DSprocessing unit 16 upon retrieval. Note that the speed of sliceretrieval may be faster retrieving slices from the slice memory 104 ascompared to retrieving the same slices from the DSN memory.

The DHT 106 lists slice name locations for slices stored in the slicememory. In another embodiment, the DHT 106 lists slice name locationsfor slices stored in at least one other cache memory.

In an example of operation, the DS processing unit 16 tracks thefrequency of retrievals of the same data object from the DSN memory 22.The DS processing unit stores the retrieved slices in the cache memory103 and updates the DHT 106 when the frequency of retrievals reaches athreshold. The DS processing unit 16 queries the DHT 106 to determine ifthe slices are stored in the cache memory 103 when receiving a retrievalrequest from a user device 1-U. The DS processing unit 16 retrieves theslices, reconstructs the data object, and sends the data object to therequesting user device when the DHT query indicates that the slices arestored in the cache memory.

In another example of operation, the DS processing unit 16 deletesslices from the cache memory when the DS processing unit 16 determinesthat the frequency of retrievals for the slices has fallen below athreshold.

The method to determine when to store slices to the cache memory 103will be discussed in greater detail with reference to FIG. 7.

FIG. 7 is a flowchart illustrating the retrieval of distributedly storeddata where the DS processing unit determines if slices are stored incache memory before retrieving the slices.

The method 700 begins with block 701, where the DS processing unitreceives a data object retrieval request from a requester (e.g., a userdevice or other system element). The request may include the data objectname and a retrieve request message. As illustrated by block 703, the DSprocessing unit updates access tracking by saving a record of theretrieval with a timestamp in the user vault or other storage area. TheDS processing unit may determine the frequency of previous retrievals byaveraging the time between the saved timestamps.

As illustrated by blocks 705 and 707, the DS processing unit determinesif the slices corresponding to the data object retrieval request are inthe cache memory by accessing the DHT and searching for the slice names.Note that DS processing unit can determine the slice names based on thedata object name as discussed previously.

As illustrated by block 709, the DS processing unit retrieves the slicesfrom the cache memory when the DS processing unit determines that theslices corresponding to the data object retrieval request are in thecache memory. The DS processing unit may verify the integrity of theslices before decoding the slices by comparing previously storedchecksums to stored checksums. As shown by block 711, the DS processingunit de-slices and decodes the slices to produce the data object inaccordance with the operational parameters as previously discussed. TheDS processing unit sends the data object to the requester, asillustrated by block 713.

As illustrated by block 715, the DS processing unit retrieves the slicesfrom the DSN memory when the DS processing unit determines that theslices corresponding to the data object retrieval request are not in thecache memory. The DS processing unit may verify the integrity of theslices before decoding the slices by comparing previously storedchecksums to stored checksums. As illustrated by block 717, the DSprocessing unit de-slices and decodes the slices to produce the dataobject in accordance with the operational parameters as previouslydiscussed.

As illustrated by blocks 719 and 721, the DS processing unit determineswhether to store the slices in the cache memory based on one or more ofa comparison of the access tracking to a threshold (e.g., the retrievalfrequency is greater than the threshold), a security level, a prioritylevel, a predetermination, and/or a network loading level. Asillustrated by block 723, the DS processing unit sends the data objectto the requester when the DS processing unit determines not to store theslices in the cache memory.

As illustrated by block 725, the DS processing unit stores the slices inthe cache memory and updates the DHT, as illustrated by block 727, withthe slice names and cache memory location when the DS processing unitdetermines to store the slices in the cache memory. As illustrated byblock 729, the DS processing unit sends the data object to therequester.

Note that the DS processing unit may determine whether to delete slicesin the cache memory based on one or more of a comparison of the accesstracking to a threshold (e.g., the retrieval frequency is less than thethreshold), the security level, the priority level, thepredetermination, and/or the network loading level. The DS processingunit deletes the slices from the cache memory and removes the slicenames from the DHT when the DS processing unit determines to deleteslices in the cache memory.

FIG. 8 is a schematic block diagram 800 of another embodiment of acomputing system where two or more DS unit storage sets are utilized toconcurrently store and retrieve EC data slices in parallel for the samedata object. As used herein the term concurrently and parallel can beconsidered interchangeable unless otherwise specified, and refergenerally to the concept of beginning storage or retrieval of one ECdata slice before a previous data slice has finished being storedretrieved.

The computing system includes a DS processing unit 16, a storage set A,and a storage set B. The DS processing unit 16 stores and retrieves ECdata slices to/from the storage sets A and B. Note that two or morestorage sets may be utilized. A storage set includes DS units thatcomprise the pillars for one or more vaults. For example, in a 4/3vault, DS units 1-4 comprise the storage set A. The corresponding secondof the two or more storage sets includes DS units 5-8 in storage set B.Both storage sets may be utilized to store slices for the same vault.Note that the two or more storage sets may be in the same or differentDSN memories.

In another embodiment of a 16/10 DSN system, storage set A includes DSunits 1-16 and storage set B includes DS units 17-32. In yet anotherembodiment, the number of DS units in the storage sets A and B aredifferent. For example, storage set A includes DS units 1-16 for a 16/10approach and storage set B includes DS units 17-20 for a 4/3 approach.

The DS processing unit 16 may determine how to implement parallelismbased on a data type, a priority level, a security level, a request, acommand, a predetermination, a desired performance level, a systemloading indicator, and/or a system configuration. For example, the DSprocessing unit 16 may utilize two storage sets when the DS processingunit determines that two storage sets will meet the desired level ofperformance (e.g., retrieval times).

To implement parallelism, the DS processing unit 16 may operate in oneof several embodiments. In a first embodiment, the DS processing unit 16creates slices for each pillar of a data segment and sends the slicesfor storage to storage set A substantially in parallel, or concurrently,with creating slices for each pillar of the next data segment andsending the slices for storage to storage set B. In other words, withtwo storage sets the DS processing unit 16 sends slices for odd datasegment numbers to storage set A while in parallel sending slices foreven data segment numbers to storage set B. The DS processing unit 16subsequently retrieves the data object by retrieving slices for eachpillar of a data segment from storage set A substantially in parallelwith retrieving slices for each pillar of the next data segment fromstorage set B.

In a second embodiment, the DS processing unit 16 creates slices foreach pillar of a series of data segments 1 through X and sends theslices for storage to storage set A substantially in parallel withcreating slices for each pillar of the next series of data segments X+1through Y and sending the slices for storage to storage set B. In otherwords, with two storage sets the DS processing unit 16 sends slices fora first series of data segment numbers to storage set A while inparallel sending slices for a second series of data segment numbers tostorage set B. The DS processing unit 16 subsequently retrieves the dataobject by retrieving slices for each pillar of a first series of datasegments from storage set A substantially in parallel with retrievingslices for each pillar of the next series of data segments from storageset B.

In a third embodiment, the DS processing unit 16 divides the data objectinto two or more sub-files, labeling them with the same filename butwith different vault generations, creating slices for each pillar of thefirst sub-file and sending the slices for storage to storage set Asubstantially in parallel, or concurrently, with creating slices foreach pillar of the next sub-file (e.g., different vault generation) andsending the slices for storage to storage set B. In other words, withtwo storage sets the DS processing unit 16 sends slices for a firstsub-file (e.g., vault gen 1) to storage set A while in parallel sendingslices for a second sub-file (e.g., vault gen 2) to storage set B. TheDS processing unit 16 subsequently retrieves the data object byretrieving slices for each pillar of the first sub-file (e.g., vault gen1) from storage set A substantially in parallel with retrieving slicesfor each pillar of the second sub-file (e.g., vault gen 2) from storageset B. The DS processing unit then combines the sub-files to recreatethe data object.

FIG. 9 is a schematic block diagram of an embodiment of a distributedstorage (DS) unit 36 that includes a storage unit control module 109 anda plurality of memories that includes memory 1 through memory m. Thestorage unit control module 109 may be implemented with the computingcore of FIG. 2. The memories may be one or more of a magnetic hard disk,NAND flash, read only memory, optical disk, and/or any other type ofread-only, or read/write memory. The memories may be implemented as partof or outside of the DS unit 36. For example, memory 1 may beimplemented in the DS unit 36 and memory 2 may be implemented in aremote server (e.g., a different DS unit operably coupled to the DS unit36 via the network).

The storage unit control module 109 may be operably coupled to thecomputing system utilizing the DSnet interface 111 via the network. Thestorage unit control module 109 may receive a EC data slice to store viathe DSnet interface 111. Note that the slice may be received as part ofa batch of slices (e.g., slices of the same pillar for the same datasegment). In an embodiment, the storage unit control module 109determines where (e.g., which address on which of the memories) to storethe received EC data slices. The determination may be based on one ormore of number of slices in the batch, slice sizes, metadata associatedwith the slices, a type of data indicator, a priority indicator,available memory, memory performance data, memory cost data, and/or anyother parameter to facilitate desired levels of efficiency andperformance.

The storage unit control module 109 may determine to utilize one or morememories 1-m for the slice batch. The storage unit control module 109may determine to evenly distribute the slice batch across the selectedmemories or the storage unit control module 109 may determine to varythe number of slices of the slice batch stored in each of the selectedmemories. For example, the storage unit control module 109 may selectmemory 2 to store all of the received slice batch since the number ofslices in the slice batch was below a threshold (e.g., a relativelysmall batch). In another example, the storage unit control module 109may select memories 1-4 to evenly distribute the received slice batchsince the number of slices in the slice batch was above a threshold(e.g., a relatively large batch). The storage unit control module 109maintains a local virtual DSN address to physical location table to keeptrack of the locations of the slices upon storage such that the slicesmay be retrieved from the proper memory upon subsequent retrievals. Inother words, the table lists the memory number and memory location foreach slice name. Note that subsequent retrievals may enjoy a morefavorable net retrieval time since memories 1-4 can simultaneouslyretrieve slices. The method to determine the memories is discussed ingreater detail with reference to FIG. 10.

FIG. 10 is a flowchart illustrating the storing of distributedly storeddata where the storage unit control module of the DS storage unitreceives slices, determines which memories to select for storage of theslices, and stores the slices in the selected memories.

The method 1010 begins at block 1013, where the storage unit controlmodule receives the slice from one or more of the DS processing unit,the storage integrity processing unit, the DS managing unit, and/or theuser device. Note that the slice may be received as part of a batch ofslices (e.g., slices of the same pillar for the same data segment). Thestorage unit control module may count the number of slices to determinethe number of slices in the slice batch. The slice may have an appendedmetadata indicating a priority, a data type, a user ID, a securitylevel, a speed of retrieval requirement, a performance requirement, areliability requirement, and/or a cost requirement.

As illustrated by blocks 1015 and 1017, the storage unit control moduledetermines a memory utilization method based on one or more of number ofslices in the slice batch, slice sizes, metadata appended and/orassociated with the slices, a performance requirement, a type of dataindicator, a priority indicator, available memory, memory performancedata, memory cost data, and/or any other parameter to facilitate desiredlevels of efficiency and performance.

In an embodiment, the storage unit control module determines the memoryutilization method to select one memory or more than one memory based inpart on the number of slices in the batch. For example, the storage unitcontrol module may select one memory when the number of slices in theslice batch is below a threshold, and more than one memory when thenumber of slices in the slice batch is above a threshold. As illustratedby block 1019, the storage unit control module stores the slices in theone memory and updates the local virtual DSN address to physicallocation table when storage unit control module determines the memoryutilization method to be one memory.

As illustrated by block 1021, the storage unit control module determinesthe distribution method when the storage unit control module determinesthe memory utilization method to be more than one memory. The storageunit control module may determine the distribution method by selectingthe number of memories based on one or more of the number of slices inthe slice batch, the slice sizes, the priority, the performancerequirements, and/or the memory performance data. For example, thestorage unit control module may select a higher number of memories whenthe performance requirements are more demanding (e.g., faster retrievaltime as compared to the average required retrieval time). The storageunit control module may select an uneven distribution of the slicesbetween the memories based on the memory performance data (e.g., actualcapabilities) of each memory. As illustrated by block 1023, the storageunit control module stores the slices substantially in parallel in thememories and updates the local virtual DSN address to physical locationtable.

Note that the storage unit control module references the local virtualDSN address to physical location table to determine which memories theslices are located in upon receiving a retrieval request from arequester (e.g., from DS processing). The storage unit control modulemay retrieve the slices substantially in parallel across two or morememories when the slices for a segment are stored in the two or morememories. The storage unit control module sends the retrieved slices tothe requester. Further note that the retrieval time performance of theDS unit may be improved when the slices are substantially retrieved inparallel from the memories.

FIG. 11 is a schematic block diagram of another embodiment of acomputing system that may provide improved security by utilizing onionrouting to communicate EC data slices.

The computing system includes the DS processing unit16, an onion layerof DS units 5-10, and a storage set 1110 of DS units 1-4. In anotherembodiment, the DS processing unit may be replaced with the DSprocessing in any one or more of the user device, the storage integrityprocessing unit, and/or the DS managing unit. The storage set 1110 mayinclude any number of DS units that comprise the pillars for one or morevaults. For example, in a 16/10 DSN system the storage set includes 16DS units while in a 4/3 DSN system the storage set includes 4 DS unitsas shown.

The DS processing unit 16 creates a layer 1 package to communicatethrough the onion layer to the storage set. The layer 1 package includesone or more of a message to be communicated to the storage set andslices to be stored in the storage set. The message may include acommand such as store, retrieve, status, and delete along with a slicename. For example, the layer 1 package may include a store command andpillar 1 slices to store in DS unit 1, pillar 2 slices to store in DSunit 2, pillar 3 slices to store in DS unit 3, and pillar 4 slices tostore in DS unit 4.

The DS processing unit 16 creates the layer 1 package based on adetermination of a number of layers of a route, and a determination ofwhich DS units (e.g., route nodes) are along the route, or in the chainof DS units. The DS processing unit 16 performs the determination of thenumber of route layers and which DS unit nodes based on one or more of asecurity requirement, a retrieval performance requirement, a randomnumber, a timer, a predetermined sequence, a user ID, a vault ID, a typeof data indicator, DS unit availability, DS unit performance history, anetwork loading indicator, a regional path requirement, and/or apriority indicator. For example, the DS processing unit may select threelayers and DS units 5, 6, and 8 to serve as the nodes in the onion layerwhen a moderate security requirement and a moderate retrievalperformance requirement is indicated. The DS processing unit may selectmore layers when the security requirement is for greater security. Notethat each of the DS units in the onion layer can also be considered topart of a chain of DS units, with the first DS unit representing thefirst layer, the first link, etc., and the end DS unit representing theinnermost onion layer, the last link in the chain, and so on.

Note that the route, or chain, may traverse any number of one or more DSunit nodes in the onion layer. The DS processing unit 16 sends the layer1 package to an entry node DS unit in the onion layer. The entry node DSunit may pass the package to an intermediate node which may pass thepackage through a series of intermediate nodes. Note that the route mayrepeat DS unit nodes. The last intermediate node may pass the package toan exit node. In an embodiment, the entry node and the exit node may bethe same DS unit (e.g., no intermediate nodes).

In another embodiment, the DS processing unit 16 may create two or morepackages with two or more selections of layers and nodes for the samedata segment or data object to send slices through the onion layer tothe storage set. In other words, the DS processing unit 16 may selectmore than one route where some of the slices are split in a first routewhile other slices traverse a different route. In an embodiment, the DSprocessing unit may send the two or more packages to two or more entrynodes as the first layer in the onion layer. For example, the DSprocessing unit 16 may send a first package of data segment 100 to entrynode DS unit 7 and a second package of data segment 100 to entry node DSunit 10. In another embodiment, the DS processing unit 16 may send thetwo or more packages bundled as one initial package to one entry node asthe first layer in the onion layer followed by an intermediate node thatmay split out the two or more packages and forward the two or morepackages to other nodes. For example, intermediate node DS unit 5 maysplit out a first package of data segment 100 and send it to exit nodeDS unit 8 and DS unit 5 may split out a second package of data segment100 and send it to exit node DS unit 10. In yet another embodiment, theintermediate node may combine packages and forward a combined package.

The DS processing unit 16 creates the layer 1 package by creating aseries of nested onion layer packages. The DS processing unit 16 startswith creating final layer package (e.g., the layer 3 package in theexample). The DS processing unit 16 creates the message for the targetstorage set (e.g., the command and/or EC data slices for storage),appending the exit node designation (e.g., DS unit 8) and encrypting allthat using the public key for the exit node (e.g., DS unit 8) to producethe layer 3 package. Next, the DS processing unit 16 creates thenext-to-last layer package (e.g., the layer 2 package in the example).In the example, the DS processing unit 16 creates the layer 2 package byappending the intermediate node designation (e.g., DS unit 5) to thelayer 3 package and encrypting all that using the public key for theintermediate node (e.g., DS unit 5) to produce the layer 2 package.Next, the DS processing unit 16 creates the entry node layer package(e.g., the layer 1 package in the example). In the example, the DSprocessing unit 16 creates the layer 1 package by appending the entrynode designation (e.g., DS unit 6) to the layer 2 package and encryptingall that using the public key for the entry node (e.g., DS unit 6) toproduce the layer 1 package. The flow described above is depictedgraphically in FIG. 12. The DS processing unit 16 method to createpackages is discussed in greater detail with reference to FIG. 13.

The DS units may store, delete, and retrieve data slices as previouslydiscussed. In an embodiment, the DS units of the onion layer may operatein accordance with one or more roles including the entry node,intermediate node, and/or exit node. The DS unit determines the rolebased on decrypting and inspecting a received package. The DS unitdecrypts the received package utilizing its private key (e.g., theprivate key is paired with the public key as utilized previously by theDS processing unit to create the package). The DS unit inspects thedecrypted package to determine if it contains the end message or aforwarding address designation (e.g., of the next node) appended to yetanother encrypted package. Note that the DS unit may not be able todecrypt the next encrypted package since that encrypted package utilizesencryption of the next node.

The DS unit sends the message to the targeted storage set when the DSunit determines its role is the exit node. The DS unit sends the messageto the next targeted onion layer node when the DS unit determines itsrole is the entry or intermediate node. The DS unit method to processpackages is discussed in greater detail with reference to FIG. 14.

FIG. 12 is a block diagram of an embodiment of layered message creationwhere the DS processing unit combines one or more of a command and/or ECdata slices from a data object into a message that is wrapped in aseries of encrypted layer packages. The graphical illustration depictsthe route example of FIG. 11 as was previously discussed.

FIG. 13 is a flowchart illustrating the creation of a layered messagewhere the DS processing unit prepares the package to send through theonion layer to the storage set.

The method begins with the step 1363, where the DS processing unitcreates the message. The message may include the command (e.g., store,retrieve, delete, status, etc.) and may include EC data slices (e.g.,created from a data object) for one or more pillars and/or supplementaryinformation (e.g., metadata about the data object).

As illustrated by block 1365, the DS processing unit determines thenumber of route layers based on one or more of a security requirement, aretrieval performance requirement, a random number, a timer, apredetermined sequence, a user ID, a vault ID, a type of data indicator,DS unit availability, DS unit performance history, a network loadingindicator, a regional path requirement, and/or a priority indicator forone or more of the other factors. For example, the DS processing unitmay select one layer when the retrieval performance requirementindicates a faster than average required retrieval time and theretrieval performance requirement has a high priority indicator.

As illustrated by block 1367, the DS processing unit determines theroute based on one or more of a security requirement, a retrievalperformance requirement, a random number, a timer, a predeterminedsequence, a user ID, a vault ID, a type of data indicator, DS unitavailability, DS unit performance history, a network loading indicator,a regional path requirement, and/or a priority indicator for one or moreof the other factors. The route may include one or more entry nodes,intermediate nodes, and exit nodes. The route may change from datasegment to data segment or for each slice. For example, the DSprocessing unit may select a route through three different geographicregions when the regional path requirement requires that the routetraverse at least three regions and the regional path requirement has ahigh priority indicator. In another embodiment, the DS processing unitmay select two routes and divide the package into two packages aspreviously discussed.

As illustrated by block 1369, the DS processing unit creates the packagestarting with the exit node. As illustrated by block 1371, the DSprocessing unit creates the package for a layer by appending the addressof the target layer node to the message (or previous package forsubsequent loops) and encrypting that together utilizing the public keyfor that layer.

As illustrated by blocks 1373 and 1375, the DS processing unitdetermines if all layers are done by comparing the just completed layerwith the entry node layer. The method branches back to block 1371, wherethe DS processing unit creating the package for a layer (the next layertowards the entry node) when the DS processing unit determines that alllayers are not done.

As illustrated by block 1377, the DS processing unit sends the packageto the entry node(s) when the DS processing unit determines that alllayers are done.

FIG. 14 is a flowchart illustrating the processing of a layered messagewhere the DS unit processes an incoming received package in accordancewith the DS unit onion layer role. The DS unit onion layer roles includethe entry node, the intermediate node, and/or the exit node. The DS unitdetermines the role based on decrypting and inspecting a receivedpackage.

As illustrated by block 1479, the DS unit receives the package from theDS processing unit or another DS unit (e.g., and intermediate node). Asillustrated by block 1481, the DS unit decrypts the received packageutilizing its private key (e.g., the private key is paired with thepublic key as utilized previously by the DS processing unit to createthe package).

As illustrated by blocks 1483 and 1485, the DS unit determines if it isthe exit node by inspecting the decrypted package. The determination maybe based on the package contents including the end message or aforwarding address designation (e.g., of the next node) appended to yetanother encrypted package.

As illustrated by block 1491, the DS unit determines the target DSunit(s) of the storage set when the DS unit determines it is the exitnode. The determination may be based on inspecting the message to readthe DSN addresses. As illustrated by block 1493, the DS unit sends themessage to the targeted DS unit(s) of the storage set.

As illustrated by block 1487, the DS unit determines the next layerdestination when the DS unit determines it is not the exit node (e.g.,it is an intermediate node or the entry node). The determination may bebased on inspecting the message to read the designation of the nextlayer node. As illustrated by block 1489, the DS unit sends the messageto the next layer node. The process repeats as described above until thepackage reaches the exit node.

In addition to the method described previously with regards to sendingan outbound message from an originator node to the endpoint distributedstorage unit as encoded multiple nested layers through a plurality ofintermediate distributed storage units, the methods described below maybe utilized to send a response message inbound from the endpointdistributed storage unit to the originator node. The method begins withthe step where a processing module of the intermediate distributedstorage unit saves the outbound information from the outbound message asit passes through the intermediate distributed storage unit.

The outbound information may include a distributed storage unitidentifier corresponding to the distributed storage unit that theoutbound message was received from, a distributed storage unitidentifier corresponding to the distributed storage unit that theoutbound message was sent to next, a message identifier, and/or adecoded key. Note that the processing module may produce the decoded keyby decrypting at least a portion of the outbound message utilizing aprivate key associated with the distributed storage unit. In someembodiments, the processing module can obtain the outbound informationfrom one or more of a lookup, a list, a predetermination a command, amessage, or another suitable source.

The method continues with the step where the processing module of theintermediate distributed storage unit receives the inbound message(e.g., a response message to a previous message). The processing moduledetermines where to forward the response message based on a responsemessage identifier, a distributed storage unit identifier correspondingto the distributed storage unit that the inbound message was receivedfrom, and the outbound information. For example, the processing moduledetermines to forward the message to DS unit 5 when the response messageidentifier correlates to a message identifier of the outboundinformation indicating that the distributed storage unit previouslyforwarded the outbound message from DS unit 5 to DS unit 2 and theinbound message was received from DS unit 2. The processing moduledetermines the decoded key based on the outbound information (e.g., thepreviously stored decoded key). The processing module encrypts at leasta portion of the inbound message utilizing the decoded key.

The above method repeats such that the inbound message may traverse aplurality of intermediate distributed storage units where each of theplurality of distributed storage units determines where to forward theinbound message, encrypts the inbound message, and forwards the inboundmessage. Some embodiments of the method end when the inbound messagereaches the originator node (e.g., a DS processing unit that sent theoriginal outbound message including each of the plurality of keysutilized by each of the intermediate distributed storage units).

The following method describes the decoding of a received inboundmessage by a processing module where the inbound message contains aplurality of layers. The processing module may be implemented in a userdevice, a DS processing unit, the storage integrity processing unit, aDS managing unit, and/or a DS unit. For example, the processing modulecan be implemented in a DS processing unit that originated an outboundmessage that corresponds to the received inbound message when theinbound message contains a response message to a message contained inthe outbound message.

The method begins with the step where the processing module receives aninbound message from an intermediate distributed storage unit. Theprocessing module determines a message to which the response messagecorresponds based on a response message identifier that can be includedwithin the inbound message, a lookup table that correlates messageidentifiers and response message identifiers, and/or the distributedstorage unit identifier of the distributed storage unit from which theinbound message was received. The processing module determines aplurality of keys and an order in which the plurality of keys is to beapplied to the message based on the message identifier and a lookuptable that correlates message identifiers with keys.

The method continues with the step where the processing module decryptsthe inbound message utilizing one of the plurality of keys in accordancewith the order determined for the plurality of keys. The method repeatsthe step to decrypt the inbound message utilizing each of the pluralityof keys in accordance with the plurality of keys order to produce anunencrypted inbound message. The processing module determines theresponse message based on the unencrypted inbound message.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “coupled to” and/or “coupling” and/or includes direct couplingbetween items and/or indirect coupling between items via an interveningitem (e.g., an item includes, but is not limited to, a component, anelement, a circuit, and/or a module) where, for indirect coupling, theintervening item does not modify the information of a signal but mayadjust its current level, voltage level, and/or power level. As mayfurther be used herein, inferred coupling (i.e., where one element iscoupled to another element by inference) includes direct and indirectcoupling between two items in the same manner as “coupled to”. As mayeven further be used herein, the term “operable to” indicates that anitem includes one or more of power connections, input(s), output(s),etc., to perform one or more its corresponding functions and may furtherinclude inferred coupling to one or more other items. As may stillfurther be used herein, the term “associated with”, includes directand/or indirect coupling of separate items and/or one item beingembedded within another item. As may be used herein, the term “comparesfavorably”, indicates that a comparison between two or more items,signals, etc., provides a desired relationship. For example, when thedesired relationship is that signal 1 has a greater magnitude thansignal 2, a favorable comparison may be achieved when the magnitude ofsignal 1 is greater than that of signal 2 or when the magnitude ofsignal 2 is less than that of signal 1.

The present invention has also been described above with the aid ofmethod steps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claimed invention.

The present invention has been described above with the aid offunctional building blocks illustrating the performance of certainsignificant functions. The boundaries of these functional buildingblocks have been arbitrarily defined for convenience of description.Alternate boundaries could be defined as long as the certain significantfunctions are appropriately performed. Similarly, flow diagram blocksmay also have been arbitrarily defined herein to illustrate certainsignificant functionality. To the extent used, the flow diagram blockboundaries and sequence could have been defined otherwise and stillperform the certain significant functionality. Such alternatedefinitions of both functional building blocks and flow diagram blocksand sequences are thus within the scope and spirit of the claimedinvention. One of average skill in the art will also recognize that thefunctional building blocks, and other illustrative blocks, modules andcomponents herein, can be implemented as illustrated or by discretecomponents, application specific integrated circuits, processorsexecuting appropriate software and the like or any combination thereof.

1. A method for use in a distributed storage processing unit, the methodcomprising: generating a first plurality of data slices includingdifferent encoded versions of a first data portion; generating a secondplurality of data slices including different encoded versions of asecond data portion; and sending the first plurality of data slices andthe second plurality of data slices to be stored concurrently indifferent storage sets, each of the different storage sets including aplurality of distributed storage units different from the distributedstorage units included in others of the different storage sets; at leasta read threshold number of data slices from a first storage set requiredto reconstruct the first data portion; and at least a read thresholdnumber of data slices from a second storage set required to reconstructthe second data portion.
 2. The method of claim 1, wherein the firstdata portion and the second data portion are data segments, the methodfurther comprising: sending data slices generated from even datasegments to be stored in the first storage set; and sending data slicesgenerated from odd data segments to be stored in the second storage set.3. The method of claim 1, wherein the first data portion and the seconddata portion are data segments, the method further comprising: sendingdata slices generated from a first sequence of data segments to bestored in the first storage set; and storing data slices generated froma second sequence of data segments to be stored in the second storageset.
 4. The method claim 1, wherein the first data portion and thesecond data portion are different vault generations of a same dataobject, the method further comprising: sending data slices generatedfrom different generations of each of a plurality of data objects to bestored in different storage sets.
 5. The method of claim 4, furthercomprising: dividing each of the plurality of data objects into aplurality of sub-files; labeling the plurality of sub-files with a samefile name, but with different vault generations; and generating thefirst plurality of data slices and the second plurality of data slicesbased, at least in part on the different vault generations.
 6. Themethod of claim 1, further comprising: determining which of a pluralityof predetermined parallel storage schemes to implement based on a systemparameter.
 7. The method of claim 6, wherein the system parameter isselected from the group consisting of data type, priority level,security level, preferred performance level, system loading indicator,and system configuration.
 8. A distributed storage processing unitcomprising: a processor to generate a first plurality of data slicesincluding different encoded versions of a first data portion and asecond plurality of data slices including different encoded versions ofa second data portion; an interface to send the first plurality of dataslices and the second plurality of data slices to be stored concurrentlyin different storage sets, each of the different storage sets includinga plurality of distributed storage units different from the distributedstorage units included in others of the different storage sets; at leasta read threshold number of data slices from a first storage set requiredto reconstruct the first data portion; and at least a read thresholdnumber of data slices from a second storage set required to reconstructthe second data portion.
 9. The distributed storage processing unit ofclaim 8, wherein the first data portion and the second data portion aredata segments, the distributed storage processing unit furthercomprising: the interface further to send data slices generated fromeven data segments to be stored in the first storage set and to senddata slices generated from odd data segments to be stored in the secondstorage set.
 10. The distributed storage processing unit of claim 8,wherein the first data portion and the second data portion are datasegments, the distributed storage processing unit further comprising:the interface further to send data slices generated from a firstsequence of data segments to be stored in the first storage set; and theinterface further to send data slices generated from a second sequenceof data segments to be stored in the second storage set.
 11. Thedistributed storage processing unit claim 8, wherein the first dataportion and the second data portion are different vault generations of asame data object, the distributed storage processing unit furthercomprising: the interface further to send data slices generated fromdifferent generations of each of a plurality of data objects to bestored in different storage sets.
 12. The distributed storage processingunit of claim 11, further comprising: the processor further to: divideeach of the plurality of data objects into a plurality of sub-files;label the plurality of sub-files with a same file name, but withdifferent vault generations; and generate the first plurality of dataslices and the second plurality of data slices based, at least in parton the different vault generations.
 13. The distributed storageprocessing unit of claim 8, further comprising: the processor further todetermine which of a plurality of predetermined parallel storage schemesto implement based on a system parameter.
 14. The distributed storageprocessing unit of claim 13, wherein the system parameter is selectedfrom the group consisting of data type, priority level, security level,preferred performance level, system loading indicator, and systemconfiguration.
 15. A distributed storage processing unit comprising: agrid module to generate a first plurality of different data slices basedon a first data segment; the grid module further to generate a secondplurality of different data slices based on a second data segment; astorage module to: assign the first plurality of different data slicesto a first storage set, the first storage set designated to be stored ina first plurality of distributed storage units; assign the secondplurality of different data slices to a second storage set, the secondstorage set designated to be stored in a second plurality of distributedstorage units different from the first plurality of distributed storageunits; an interface to send the second plurality of different dataslices to be stored substantially in parallel with sending the firstplurality of different data slices to be stored.
 16. The distributedstorage processing unit of claim 15, further comprising: the storagemodule to assign data slices generated from even data segments to thefirst storage set, and to assign data slices generated from odd datasegments to the second storage set.
 17. The distributed storageprocessing unit of claim 15, further comprising: the storage module toassign data slices generated from a first sequence of data segments tothe first storage set, and to assign data slices generated from a secondsequence of data segments to the second storage set.
 18. The distributedstorage processing unit claim 15, wherein the first data segment and thesecond data segment are sub-segments of a same data segment, thedistributed storage processing unit further comprising: the grid moduleto generate slice names of the first data segment and the second datasegment, the slice names including different vault generations; and thestorage module further to assign data slices having slice names thatinclude different vault generations to different storage sets.
 19. Thedistributed storage processing unit of claim 18, further comprising: anaccess module to determine a vault generation associated with each of aplurality of sub-files having a same file name; and the grid module togenerate the first plurality of different data slices and the secondplurality of different data slices based, at least in part on the vaultgeneration.
 20. The distributed storage processing unit of claim 15,further comprising: the storage module further to determine which of aplurality of predetermined parallel storage schemes to implement basedon a system parameter.
 21. The distributed storage processing unit ofclaim 20, wherein the system parameter is selected from the groupconsisting of data type, priority level, security level, preferredperformance level, system loading indicator, and system configuration.